Efficient allocation of network resources, such as available network bandwidth, has become critical as enterprises increase reliance on distributed computing environments and wide area computer networks to accomplish critical tasks. The widely-used TCP/IP protocol suite, which implements the world-wide data communications network environment called the Internet and is employed in many local area networks, omits explicit supervisory function over the rate of data transport over the various devices that comprise the network. While there are certain perceived advantages, this characteristic has the consequence of juxtaposing very high-speed packets and very low-speed packets in potential conflict and produces certain inefficiencies. Certain loading conditions degrade performance of networked applications and can even cause instabilities which could lead to overloads that could stop data transfer temporarily. The above-identified U.S. Patents and patent applications provide explanations of certain technical aspects of a packet based telecommunications network environment, such as Internet/Intranet technology based largely on the TCP/IP protocol suite, and describe the deployment of bandwidth management solutions to monitor and/or manage network environments using such protocols and technologies.
Effective management of such networks often requires regular monitoring and collection of data characterizing various attributes of the network, its operation and/or the traffic flowing through it. For example, Cisco Systems, Inc. of San Jose, Calif. offers a feature set of data monitoring and collection technologies in connection with its routers, called Netflow®. The Cisco IOS® NetFlow feature set allows for the tracking of individual IP flows as they are received at a router or switching device. According to the technology, after a flow has terminated, a suitably configured router or switch generates a NetFlow record characterizing various attributes of the flow. The NetFlow record is ultimately transmitted as a datagram to a NetFlow Data Collector that stores and, optionally, filters the record. A NetFlow Record includes a variety of attributes, such as source and destination IP addresses, packet count, byte count, start and end time stamps, source and destination TCP/UDP ports, Quality of Service attributes, and routing-related information (e.g., nexthop and Autonomous System (AS) data). Such NetFlow® records are similar to call records, which are generated after the termination of telephone calls and used by the telephone industry as the basis of billing for long distance calls, for example.
Furthermore, most network devices maintain data characterizing utilization, operation and/or performance of the network devices, and/or the network on which the devices operate, in limited, volatile memory, rather than using persistent storage (e.g., hard disks or other non-volatile memory). Consequently, network management applications commonly use the Simple Network Management Protocol (SNMP) to poll network devices (using the Management Information Base (MIB) associated with the network device) at regular time intervals and maintain the sampled raw data in a persistent data store. The network management application, such as a reporting package, then processes the raw data to allow for the creation of reports derived from the raw data detailing operation and/or performance of the device and/or the network. Management Information Bases typically contain low-level information characterizing the operation of the network device, such as the number of bytes or packets encountered on an interface, and do not provide information concerning the characteristics of data flows. Using a reporting package, a network administrator may then analyze the data to yield information about the performance or utilization of the network and/or network devices associated with the network. Indeed, various applications can then access and analyze the collected data for a variety of purposes, including accounting, billing, network planning, traffic engineering, and user or application monitoring.
Packeteer, Inc. of Cupertino, Calif. develops application traffic monitoring, management, and reporting software and systems. Its PacketSeeker® systems and PacketShaper® application traffic management devices, among other things, provide flow- and application-aware monitoring of network traffic enabling classification of network traffic flows on a per-application basis. The Packetshaper® application traffic management device, for example, includes functionality allowing for classification of network traffic based on information from layers 2 to 7 of the OSI reference model. As discussed in the above-identified patents and patent applications, the application traffic management device may include a measurement engine operative to record or maintain numeric totals of a particular measurement variable at periodic intervals on a per-application or other traffic classification basis. The application traffic management device further includes a management information base including standard network objects maintaining counts relating, for example, to the operation of its network interfaces and processors. Packeteer's ReportCenter™ leverages the powerful network utilization and application performance statistics available in Packetshaper® bandwidth management devices and offers a centralized reporting platform to monitor and manage large deployments efficiently by streamlining collection, collation, storage, analysis, and distribution of measured statistics.
As described above, the measurement engine operates similarly to a standard management information base in that various statistics and counters are maintained for a plurality of traffic classes or types. For example, the measurement engine can record the number of inbound and outbound packets, as well as the number of bytes, traversing a network device on an aggregate, application, and/or traffic class level. Other network statistics can include the number of TCP packets, the number of retransmitted TCP packets, the peak number of active TCP flows or other connections, etc. The measurement engine may also maintain data relating to configuration of an application traffic management device, such as the number of bandwidth partitions, the byte count in a given partition, the packet count in a given partition, the TCP data packet count in a given partition, the TCP retransmit packet count in a given partition, the TCP tossed retransmit packet count in a given partition, the peak number of active TCP flows in the partition, the total time in seconds spent over the partition size for the partition. The measurement engine may also maintain data relating to traffic classes, such as, for a given traffic class: the packet count in the traffic class, the TCP data packet count in the class, the TCP retransmit packet count in the class, and the peak number of active TCP flows in the class. In most implementations, the measurement engine maintains such values in volatile memory (in a table or other data structure) and, at periodic intervals, stores the data in persistent memory, such as a hard drive, with a time stamp and clears the network statistic values in the volatile memory space.
While the measurement engine is sufficient to achieve its intended purpose, the granularity of the data maintained by the measurement engine is generally limited to traffic classes and partitions. Often times, however, it is desirable to obtain a more granular view of the network traffic. For example, the measurement engine does not allow for isolation of specific hosts on the network. Furthermore, the table or other data structure used to record the measurement engine data is relatively static. Adding a new basis of classification to the measurement engine, therefore, would require reconfiguring the cache table of the measurement engine and a loss of the data previously maintained. This circumstance, therefore, limits the application of dynamic or adaptive measurement policies that could be implemented in response to a detected event.
In light of the foregoing, a need in the art exists for methods, apparatuses and systems that allow for, or facilitate, dynamic or adaptive measurement systems. In addition, a need exists for methods, apparatuses and systems that allow for collection of more granular network traffic data. Embodiments of the present invention substantially fulfill these needs.